# Zero-Shot Learning

VideoGrain is a diffusion model-based video editing technology that achieves multi-granularity video editing by adjusting the spatiotemporal attention mechanism. This technology addresses the issues of semantic alignment and feature coupling in traditional methods, enabling fine-grained control over video content. Its key advantages include zero-shot editing capabilities, efficient text-to-region control, and feature separation capabilities. This technology is suitable for scenarios requiring complex video editing, such as post-production in film and television and advertising production, significantly improving editing efficiency and quality.

X-Dyna is an innovative zero-shot human image animation generation technology that creates realistic and expressive dynamic effects by transferring facial expressions and body movements from a driving video to a single human image. This technology uses a diffusion model and integrates reference appearance context effectively into the diffusion model's spatial attention through the Dynamics-Adapter module, while maintaining the motion module's ability to synthesize smooth and complex dynamic details. It not only enables body posture control but also captures and precisely transfers identity-independent facial expressions through a local control module. Trained on mixed datasets of various human and scene videos, X-Dyna can learn physical human motions and natural scene dynamics, generating highly realistic and expressive animations.

SAMURAI is a visual object tracking model based on the Segment Anything Model 2 (SAM 2), specifically designed to handle the visual tracking task of fast-moving or self-occluding objects. It effectively predicts object motion and optimizes mask selection by introducing temporal motion cues and a motion-aware memory selection mechanism, achieving robust and accurate tracking without the need for retraining or fine-tuning. SAMURAI operates in real-time environments and demonstrates strong zero-shot performance across multiple benchmark datasets, proving its generalization capability without the need for fine-tuning. In evaluations, SAMURAI showed significant improvements in success rate and accuracy compared to existing trackers, with a 7.1% AUC increase on LaSOT-ext and a 3.5% AO increase on GOT-10k. Furthermore, compared to fully supervised methods on LaSOT, SAMURAI demonstrates competitive results, underscoring its robustness in complex tracking scenarios and potential practical value in dynamic environments.

PromptFix is a comprehensive framework that enables diffusion models to perform various image processing tasks by following human instructions. This framework constructs a large-scale instruction-following dataset, proposes high-frequency guided sampling methods to control the denoising process while retaining high-frequency details in unprocessed areas, and designs auxiliary prompt adapters that enhance text prompts using visual language models, increasing the model's task generalization capabilities. PromptFix outperforms previous methods across various image processing tasks and demonstrates superior zero-shot capabilities in blind restoration and compositional tasks.

ROCKET-1 is a Visual-Language Model (VLM) specifically designed for embodied decision-making in open-world environments. This model connects VLMs with policy models through a visual-temporal context prompting protocol, guiding policy-environment interactions using object segmentation from past and current observations. By this means, ROCKET-1 unlocks the visual-language reasoning capabilities of VLMs, enabling it to solve complex creative tasks, especially in spatial understanding. Experiments with ROCKET-1 in Minecraft demonstrate that this approach allows agents to accomplish previously unattainable tasks, highlighting the effectiveness of visual-temporal context prompting in embodied decision-making.

Prompt Engineering is a cutting-edge technology in the field of artificial intelligence that is transforming how we interact with AI technologies. This open-source project aims to provide a platform for both beginners and seasoned practitioners to learn, build, and share Prompt Engineering techniques. The project includes a variety of examples ranging from basic to advanced levels, aimed at fostering learning, experimentation, and innovation in the field of Prompt Engineering. Additionally, it encourages community members to share their innovative techniques, collectively advancing the development of Prompt Engineering.

Seed-Music is a music generation system that supports the creation of expressive multilingual vocal music within a unified framework, allowing for precise note-level adjustments and the integration of users' voices into music compositions. The system employs advanced language models and diffusion models, providing diverse creative tools to meet various music production needs.

MimicBrush is an innovative image editing model that allows users to achieve zero-shot image editing by specifying the editing area in the source image and providing a reference image. The model can automatically capture the semantic correspondence between the two images and complete the editing in one step. MimicBrush is developed based on diffusion priors, capturing semantic relationships between different images through self-supervised learning, and experimental results demonstrate its effectiveness and superiority in various test cases.

Slicedit is a zero-shot video editing technique that utilizes text-to-image diffusion models coupled with spatiotemporal slicing to enhance temporal consistency in video editing. This technology preserves the original video's structure and motion while adhering to the target text description. Extensive experiments have demonstrated Slicedit's significant advantages in editing real-world videos.

NaturalSpeech 3 aims to enhance speech synthesis quality, similarity, and rhythm by decomposing the various attributes of speech (e.g., content, prosody, timbre, and acoustic details) and generating each attribute separately. The system designs a neural encoder-decoder with decomposed vector quantization (FVQ) to decouple the speech waveform and proposes a decomposed diffusion model to generate each sub-space attribute based on corresponding prompts.

Cola is a method that uses a language model (LM) to aggregate the outputs of 2 or more vision-language models (VLMs). Our model assembly method is called Cola (COordinative LAnguage model or visual reasoning). Cola performs best when the LM is fine-tuned (called Cola-FT). Cola is also effective in zero-shot or few-shot context learning (called Cola-Zero). In addition to performance improvements, Cola is also more robust to VLM errors. We demonstrate that Cola can be applied to various VLMs (including large multimodal models like InstructBLIP) and 7 datasets (VQA v2, OK-VQA, A-OKVQA, e-SNLI-VE, VSR, CLEVR, GQA), and it consistently improves performance.

DirectAI is a platform based on large language models and zero-shot learning that allows you to build models tailored to your needs instantly based on your description, without requiring training data. You can deploy and iterate on models within seconds, saving you the time and expense of assembling, labeling training data, training models, and fine-tuning them. Headquartered in New York City, DirectAI, backed by venture capital, is transforming the way people use artificial intelligence in the real world.